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ABSTRACT 

Biological  evidence  suggests  that  information 
encoding  in  the  form  of  oscillatory  patterns  is 
advantageous  compared  to  convergent,  fixed-point 
type  memories.  Freeman’s  Kill  model  is  an  example 
of  operational  chaotic  memory  neural  networks. 
Noise  plays  a  constructive  role  in  the  model  by 
maintaining  and  stabilizing  aperiodic  orbits.  Gaussian 
noise  components  are  injected  to  the  model  at 
different  locations;  at  the  input  channels  and  also  at  a 
centrally  located  internal  node.  Depending  on  the 
noise  intensity  and  bias,  resonance  effects  have  been 
identified  in  the  Kill  model.  The  observed  noise 
effects  have  some  similarity  with  stochastic 
resonance,  but  there  are  very  essential  differences. 
The  interaction  of  noise  with  the  oscillatory  signal 
has  a  resonance  character  in  the  Kill  model.  The 
oscillatory  signal  in  Kill,  however,  is  not  coming 
from  the  external  world,  but  it  is  the  result  of  the 
interaction  of  the  various  internal  components. 
Therefore,  the  signal  has  an  intimate  interference 
with  the  noise.  These  effects  are  illustrated  in  pattern 
recognition  problems. 

Keywords:  Chaotic  Neural  Memory,  Noise,  Freeman 
Network,  Pattern  Recognition. 

1.  INTRODUCTION  TO  CONVERGENT  AND 
OSCILLATORY  NEURAL  NETWORKS 

An  overwhelming  part  of  neural  networks  today  uses 
fixed  point  convergence  for  information  encoding. 
The  reason  of  the  wide  proliferation  of  convergent 
NNs  is  their  relative  simplicity,  the  existence  of  firm 
mathematical  foundations  established  during  a  latent 
period  in  the  60  s  and  70  s,  and  the  explosive 
development  of  this  field  in  the  80  s.  The  past  decade 
showed  a  wide  range  of  impressive  applications  of 
NNs  together  with  other  connection! st,  soft 
computing  technologies  in  the  field  of,  e.g.,  pattern 


recognition,  system  identification,  and  adaptive 
control,  see  Schwartz  (1990). 

There  is  a  growing  number  of  neural  network  models 
that  use  nonlinear  dynamics  for  information 
processing.  Often  these  models  use  limit  cycle 
oscillations  for  encoding  and  retrieving  data  Fukai  Sc 
Shiino  (1992),  Wang  (1998).  Using  bifurcation 
sequences  of  an  attractor  in  a  high-dimensional  space 
can  serve  to  that  purpose.  In  equilibrium  NNs,  the 
stored  memory  patterns  correspond  to  local  minima 
of  the  static  landscape  of  the  network  energy 
function.  The  shape  of  this  energy  function  can  be 
very  complex  and  it  may  contain  a  lot  of  spurious 
local  minima  preventing  the  correct  retrieval  of 
certain  patterns.  Limit  cycle-based  NNs  do  not  suffer 
from  this  problem  as  they  encode  the  patterns  not  in  a 
single  minimum,  but  in  a  finite  sequence  of  jumps. 

Another  advantage  of  limit  cycle  encoding  is  memory 
efficacy.  The  number  of  patterns  memorized  by 
equilibrium  NNs  cannot  exceed  the  number  of  fixed 
points  of  the  dynamics,  which,  in  turn,  are  strongly 
limited  as  the  function  of  the  number  of  processing 
elements.  Bifurcative  NN  encoding  schemes  have 
higher  memory  capacity  by  realizing  an  encoding  as 
a  combination  of  jumps  among  the  fixed  points  of  the 
dynamics.  As  a  whole,  however,  the  memory  is  still 
linked  to  the  fixed  points  of  the  dynamics  in  both 
equilibrium  and  bifurcative  schemes. 

In  typical  equilibrium  type  encoding  systems,  the 
noise  and  the  overlap  of  the  pattern  classes  is 
undesirable.  In  non-equilibrium  NNs,  the  noise  can 
play  a  completely  different,  constructive  role. 
Examples  are  the  noise-mediated  signal  enhancement 
in  stochastic  resonance  (SR)  Gammiatoni  et  al. 
(1998)  and  the  stabilization  of  aperiodic  attractors  by 
sensory  and  central  noise  in  the  Freeman  Kill  sets; 
Freeman  (1994). 


excitatory  (E)  or  inhibitoiy  (I)  KO  sets,  Ki_E  or  KIJ 
sets  are  formed,  respectively.  The  interaction  of  KI_E 
and  KI_I  sets  constitutes  the  KII  set.  Finally, 
coupling  several  KII  sets  with  excitatory,  inhibitory 
feed-forward  and  feedback  loops,  one  arrives  at  the 
Kill  set. 

The  Kill  model  is  an  example  of  layered  networks 
with  nonlinear  units  having  the  following  types  of 
coupling:  (i)  feed-forward  connections  between 
layers,  (ii)  lateral  excitation  or  inhibition  across 
certain  layers,  (iii)  feedback  connections  between 
layers.  Decade-long  studies  indicate  that  this  class  of 
networks  can  exhibit  a  wide  range  of  dynamic 
behaviors,  i.e.,  fixed  point  and  limit  cycle  attractors, 
quasi-periodic  oscillations  and  chaos.  Kill  models 
can  grasp  the  essence  of  the  observed  dynamic 
behavior  in  certain  biological  neural  networks, 
including  that  of  the  olfactory  system,  Freeman 
(1992).  In  the  Kill  model  of  the  olfaction  the  layers 
correspond  to:  receptors  (R),  periglomerular  cells  (PI 
and  P2),  olfactory  bulb  (OB),  anterior  olfactory 
nucleus  (AON),  prepyriform  cortex  (PC)  and  deep 
pyramidal  cells  (C).  There  is  a  general  feed-forward 
structure  from  R  to  PI  and  OB,  and  from  OB  to  AON 
and  PC  via  the  lateral  olfactory  tract  (LOT).  Lateral 
connections  are  incorporated  at  the  OB  at  two  levels, 
while  feedback  is  directed  from  PC  to  AON,  from  C 
to  OB,  and  from  AON  to  OB  and  PI  via  the  medial 
olfactory  tract  (MOT).  The  OB,  the  AON  and  PC  are 
all  examples  of  interconnected  KII  sets. 

The  operation  of  the  memory  can  be  described  as 
follows.  In  the  absence  of  stimuli  the  system  is  in  a 
high  dimensional  state  of  spatially  coherent  basal 
activity.  The  basal  state  is  described  by  an  aperiodic 
(chaotic)  global  attractor.  In  response  to  external 
stimulus,  the  system  can  be  kicked-off  from  the  basal 
state  into  a  local  memory  wing.  This  wing  is  usually 
of  much  smaller  dimension  than  the  basal  state.  It 
shows  coherent  and  spatially  patterned  amplitude- 
modulated  (AM)  fluctuations.  After  the  residence  of 
the  system  in  this  localized  wing,  it  returns  to  the 
basal  state.  This  is  a  temporal  burst  process  having  a 
duration  of  about  100  ms.  A  memory  pattern  is 
defined  therefore  as  a  spatio-temporal  process 
represented  by  the  sequence  of  spatial  AM  patterns 
during  the  burst. 

The  Kill  model  has  been  used  to  actually  implement 
the  above  memory  process.  Using  advanced 
optimization  techniques,  the  system  can  be  trained  to 
learn  a  collection  of  given  patterns  (Chang  and 
Freeman,  1996),  There  is  a  problem,  however,  in  the 
utilization  of  the  Kill  model  as  a  powerful  dynamical 
memory  device.  It  is  the  fragmentation  of  the  global 


aperiodic  (chaotic)  attractor  into  quasi-periodic  local 
attractors,  following  input-induced  state  transitions. 
The  optimized  system  is  extremely  sensitive  to  small 
changes  in  system  parameters  due  to  attractor 
crowding  (Chang  et  al.,  1998)  which  practically 
prevented  effective  generalization.  This  problem  has 
been  the  impediment  of  further  progress  in  this  field 
for  several  years. 

Attractor  crowding  is  an  unavoidable  manifestation 
of  the  complexity  of  the  high-dimensional  chaotic 
Kill  system.  Highly  evolved,  fractured  attractor 
boundaries  are  found,  which  produce  a  mixture  of 
various  attractors  in  a  small  neighborhood  of  a 
typical  point  of  the  state  space.  As  the  external 
conditions  vary,  the  low-dimensional  subspace  into 
which  the  system  collapses  also  changes.  The 
complicated  landscape  indicates  that  fine-tuning  the 
system  parameters  to  a  narrow  section  of  the  given 
attractor  could  be  a  daunting  and  practically 
unfeasible  task.  As  an  Important  development  in  the 
theory  and  implementation  of  dynamical  memories,  it 
has  been  shown  that  attractor  fragmentation  can  be 
effectively  utilized  in  building  robust  memories 
Kozma  and  Freeman  (1999,  2000),  Accordingly,  we 
acknowledge  the  co-existence  of  a  range  of  attractors 
in  any  actual  realization  of  the  system  and  build 
robust  Kill  dynamics  in  this  way. 

The  main  idea  is  illustrated  on  the  example  of  the 
relatively  simple  KII  subsets.  Various  KII  set  can  be 
identified  in  the  olfactory  system,  as  OB,  AON,  and 
PC,  The  impulse  responses  of  these  KII  sets  are 
decayed  oscillations.  There  are,  however,  important 
differences  in  the  asymptotic  behaviors. 
Physiological  evidence  shows  unbiased  fixed  point 
for  the  OB,  and  inhibitory  and  excitatory  bias  for  PC 
and  AON,  respectively;  see  Freeman  (1992). 
Parameter  studies  show  that  there  are  significant 
overlaps  among  the  attractors,  depending  on  the 
initial  conditions.  This  behavior  resembles  the 
partially  ordered  phase-type  attractors  (glassy  or 
intermittent)  in  globally  coupled  lattices,  Kaneko 
(1990).  Details  of  the  dynamical  behavior  of  KII  sets 
are  given  in  Kozma  and  Freeman  (2000).  In  this 
paper,  we  concentrate  on  the  highest  level  dynamics 
of  the  Kill  model. 

4.  PATTERN  RECOGNITION  USING 
SPACE-TIME  ENCODING 

There  are  several  requirements  toward  a  biologically 
plausible  model  of  chaotic  memories.  Remaining 
with  the  example  of  the  olfaction,  these  conditions 
can  be  formulated  as  follows.  Freeman  et  al.  (1997): 


•  broad  band  aperiodic  fluctuations  in  the  basal 
state  of  the  olfactory  bulb  with  1/f-  type  power 
spectral  densities , 

•  spatially  coherent  behavior  in  the  olfactory  bulb, 

•  temporal  fluctuations  at  various  levels  should 
have  close  to  normal  distribution  in  the  basal 
state, 

•  the  magnitudes  of  the  wave  density  signals  of  the 
periglomerular  (PG)  cells  should  concentrate  on 
the  right  of  the  maximum  gradient  of  the 
asymmetric  wave-to-pulse  transfer  function, 

•  the  magnitudes  of  the  wave  densities  of  the 
signals  in  the  olfactory  bulb  should  concentrate 
on  the  left  of  the  maximum  derivative  of  the 
transfer  function, 

•  phase  transition  should  occur  swiftly,  within 
about  10  ms  after  the  onset/termination  of  the 
stimuli* 

The  Kill  model  satisfies  these  conditions  and  it  can 
be  considered  as  a  biologically  realistic  model  of 
measured  EEG  signals. 

In  the  Kill  model,  several  learning  tools  are  used. 
One  is  the  Hebbian  learning  of  stimulus  patterns,  the 
other  is  habituation  of  background  activity. 
Habituation  can  be  modeled  as  weight  decay  in  the 
form  of  forgetting,  as  it  is  described  in  the  previous 
sections.  A  third  mechanism  involves  nonlinear 
adaptive  control  techniques  aiming  at  the 
stabilization  of  the  trajectories.  All  these  learning 
methods  exist  in  a  subtle  balance  and  their  relative 
importance  changes  at  various  stages.  As  a  read-out 
procedure  from  the  Kill  model,  AM  patterns  are 
calculated  for  the  BxSlattice  on  the  Mitral  level.. 

The  Kill  model  is  used  for  the  classification  of  the 
experimentally  observed  RMS  data.  Experiments 
have  been  conducted  with  rabbits  which  were 
chronically  implanted  with  square  arrays  of  64  (8x8) 
electrodes.  The  rabbits  have  been  trained  to  recognize 
two  types  of  visual  stimuli.  The  measured  data  in  the 
form  of  64-dimensional  patterns  have  been  used  as 
input  patterns  to  the  Kill  model.  Readout  is 
conducted  at  the  mitral  level  (3rd  layer)  of  Kill  in  the 
form  of  gamma-band  (4GHz  to  80  Hz)  oscillation 
intensity. 

Examples  of  the  EEG  rms  patterns  are  shown  in  Fig. 
1.  One  can  see  clear  differences  in  the  rms  patterns 
corresponding  to  the  different  stimuli.  At  different 
time  steps,  the  differences  can  become  less  obvious 
or  even  confusing,  as  it  is  seem  in  Fig.  1 .  The  change 
of  the  spatial  AM  magnitude  pattern  during  the  input 
induced  phase  transition  is  clearly  visible.  However, 
there  is  no  ’fixed’  pattern  of  rms  for  identification. 


Rather  we  have  to  rely  of  the  sequence  of  somewhat 
blurred  patterns  with  intermittent  fluctuations  In  our 
classification  task. 
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Figure  1  Spatio-temporal  sequence  of  AM  patterns;  post- 
stimulus  signal  analysis  of  rabbit  experiments  with  two 
visual  stimuli:  A  and  B.  The  time  step  of  the  image 
processing  is  64  ms;  sampling  time  is  1  ms. 

We  used  5  patterns  from  both  classes  for  the  training 
in  10  sequential  steps.  Consecutively,  the  trained 
model  has  been  used  for  classifying  30  untrained 
patterns  (15  from  each  class).  In  the  classification  we 
have  evaluated  the  Euclidean  distance  between  a 
given  pattern  and  the  previously  determined  centers 
of  the  two  classes.  We  evaluated  these  distances  for 
the  RMS  patterns  calculated  on  the  OB  level  in  the 
model  and  compared  it  with  the  classification  based 
on  the  distances  obtained  for  the  original  input  data 
sets. 

100  • 
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Figure  2:  Classification  performance  of  Kill  (solid  line); 
the  classification  performance  of  the  pure  statistical 
clustering  method  is  shown  by  dash.  Note  the  significant 
improvement  of  the  performance  of  the  model  at  moderate 
additive  noise  levels. 

The  results  are  summarized  in  Figure  2,  where  the 
correct  classification  rate  of  the  Kill  model  is  shown 
in  solid  line.  The  classification  rate  based  on  the 


original  input  EE G  rms  data  (without  noise)  is  given 
in  dashed  line.  It  is  seen  that,  by  applying  moderate 
additive  noise  levels,  we  could  achieve  an  improved 
classification  performance  by  Kill  compared  to  the 
pure  clustering  method. 

5,  ROLE  OF  NOISE  IN  THE  MEMORIES 

In  this  section,  the  relation  between  the  aperiodic 
behavior  of  the  Kill  and  typical  low-dimensional 
deterministic  chaos  is  discussed.  Also,  we  explain  the 
novel  features  of  parametric  noise  effect  in  the  Kill 
model  in  the  context  of  stochastic  resonance  (SR).  SR 
is  another  interesting  application  of  information 
processing  in  noisy  chaotic  systems.  SR  has  become 
a  well-established  research  field  during  the  past  two 
decades  and  it  is  widely  applied  in  various 
disciplines,  from  laser  physics,  semiconductor 
devices,  through  neurophysiology.  SR  effect  can 
arise  in  a  bi-  or  multi -stable  system  with  an  energy 
threshold  between  the  states.  External  or  internal 
noise  can  initiate  a  transition  between  the  states.  This 
effect  has  a  resonance  character  and  it  can  be  used  to 
enhance  a  weak  (periodic)  input  signal,  thus 
producing  a  high  signal-to-noise  ratio.  For  an 
overview  of  this  field,  see  Gammaitoni  et  al.  (1998), 
Assumian  &  Moss  (1998).  Examples  of  successful 
implementation  in  neural  systems  are  Moss  &  Pei 
(1995),  Bulsara  et  al.  (1996),  Mitaim  &  Kosko 
(1998),  among  others. 

It  is  very  likely  that  brains  use  resonance  effects  in  a 
more  subtle  way  than  it  is  suggested  by  the  original 
SR  theory.  This  issue  will  be  elaborated  further  in 
connection  with  the  properties  of  the  Freeman  Kill 
nets.  Noise  has  a  special  role  in  the  Kill  model. 
There  are  Gaussian  noise  components  injected  at 
different  locations:  at  the  input  channels  and  also 
centrally  at  the  AON.  The  input  noise  is  spatially 
independent  and  rectified  while  the  central  noise  in 
centrifugal  and  uniform  in  space  with  possible  bias. 
Depending  on  the  noise  intensity  and  bias,  resonance 
effects  have  been  identified  in  the  Kill  model. 

Brain  chaos  has  been  analyzed  experimentally  and 
theoretically  and  it  is  called  stochastic  chaos  by 
Freeman  (2000).  These  noise  effect  have  some 
similarity  with  stochastic  resonance  but  they  have 
very  crucial  differences  as  well  (Kozma  &  Freeman, 
1999).  A  comparison  of  SR  and  Kill  resonance  is 
given  in  Table  I.  SR  has  3  main  components:  a  bi-  or 
multi-stable  energy  function,  weak  (periodic)  input 
signal  and  noise.  The  addition  of  noise  can  enhance 
the  signal-to-noise  ratio  which  is  of  great  practical 
importance  for  signal  processing  applications.  The 
interaction  of  noise  with  the  oscillatory  signal  has  a 


resonance  character  in  the  Kill  model  as  well.  The 
oscillatory  signal  in  the  Kill  model,  however,  is  not 
coming  from  the  external  world,  but  it  is  the  result  of 
the  interaction  of  the  various  internal  KII 
components.  Therefore,  the  signal  may  have  an 
intimate  interference  with  the  noise  in  a  Kill  system, 
as  compared  with  the  pure  input/output  relationship 
in  the  case  of  SR.  Another  difference  is  that  the 
energy  function  is  fixed  In  SR  while  the  complex 
energy  function  can  be  adapted  In  the  Kill  model. 

6.  CONCLUDING  REMARKS 

The  Kill  model  produces  biologically  plausible 
results  that  are  used  for  the  Interpretation  of  EEG 
measurements  conducted  in  mammalian  olfactory 
systems.  Intensive  future  studies  are  required  to 
discover  the  complexity  of  the  dynamics  of  rules 
inferred  from  various  neural  networks  and  to  utilize 
them  in  understanding  Intelligence  In  computational 
and  biological  systems. 
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Table  I  Stochastic  Resonance  versus  Parametric  Noise  in  Kill 


Stochastic  Resonance  (SR) 

Parametric  effect  of  noise  in  Kill 

Fixed  bi-  (multi-)  stable  nonlinear  system 

Continuously  changing  multi-stable  nonlinear  system 

Weak  (periodic)  signal  transmitted 

The  fluctuating  carrier  signal  originates  from  inside 

External  and  internal  noise  magnifies  the  input  signal  by 
de-stabilizing  chaos 

Input  and  central  noise  stabilizes  the  system  and  amplifies 
the  chaotic  signals 

Maximum  signal-to-noise  ratio  at  a  well-defined  noise 
level  (resonance) 

Maximum  amplification  at  some  intermediate  noise  level 

Components  are  chaotic 

Components  are  not  chaotic  and  chaos  is  the  collective 
feature  of  Kill 

